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METHOD OF VIDEO CODING THE MOVEMENT OF A HUMAN FACE FROM A SEQUENCE 

OF IMAGES 

RELATED APPLICATIONS 

This patent application is related to concurrently filed US Patent Application Serial No. 

, titled "Model-Based Video Image Coding," by Acharya et al., filed on , 

(attorney docket 042390. P8764), and concurrently filed US Patent Application Serial No. 

, titled "Method of Video Coding Shoulder Movement from a Sequence of 

Images," by Acharya et al., filed on , (attorney docket 042390. P8763), both assigned in 

part to the assignee of the present invention and herein incorporated by reference. 

BACKGROUND 



The present disclosure is related to video coding and, more particularly, to coding the 
movement of a head from a sequence of images. 



As is well-known, motion estimation is a common or frequently encountered problem in 
digital video processing. A number of approaches are known and have been employed. One 
approach, for example, identifies the features located on the object and tracks the features from 
frame to frame, as described for example in "Two-View Facial Movement Estimation" by H. Li 
and R. Forchheimer, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 4, 
No. 3, pp. 276-287, June, 1994. In this approach, the features are tracked from the two- 
dimensional correspondence between successive frames. From this correspondence, the 
three-dimensional motion parameters are estimated. Another approach estimates the motion 
parameters from an optical flow and affine motion model. See, for example, "Analysis and 
Synthesis of Facial Image Sequences in Model-Based Coding," by C.S. Choi, K. Aizawa, H. 
Harashima and T. Takeve, IEEE Transactions on Circuits and Systems for Video Technology, 
Vol. 4, No. 3, pp. 257-275, June, 1994. This optical flow approach estimates the motion 
parameters without establishing a two-dimensional correspondence. This latter approach, 
therefore, tends to be more robust and accurate, but imposes a computational load that is 
heavier typically. A need, therefore, exists for an approach that is more accurate then the two- 
dimensional correspondence approach, but that is computationally less burdensome than the 
optical flow and affine motion model. 



BRIEF DESCRIPTION OF THE DRAWINGS 



The subject matter regarded as the invention is particularly pointed out and distinctly 
claimed in the concluding portion of the specification. The invention, however, both as to 
organization and method of operation, together with objects, features, and advantages thereof, 
may best be understood by reference to the following detailed description when read with the 
accompanying drawings in which: 

FIG. 1 is a schematic diagram illustrating a three-dimensional (3D) model applied to a 
human face with planar triangular patches, such as may be employed in an embodiment in 
accordance with the present invention; 

FIG. 2 is a flowchart of an embodiment in accordance with the present invention. 
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DETAILED DESCRIPTION 



In the following detailed description, numerous specific details are set forth in order to 
provide a thorough understanding of the invention. However, it will be understood by those 
skilled in the art that the present invention may be practiced without these specific details. In 
other instances, well-known methods, procedures, components and circuits have not been 
described in detail so as not to obscure the present invention. 

As previously described, motion estimation is a common problem in video image 
processing. However, state of the art techniques such as previously described, for example, 
suffer from some disadvantages. For example, the previously described technique, referred to 
here as the "two-dimensional correspondence approach," although computationally less 
burdensome, seems to be prone to errors due to mismatches of the two-dimensional 
correspondences. Another approach, referred to here as the "optical flow and affine motion 
model," such as described in "3-D Motion Estimation and Wireframe Adaptation Including 
Photometric Effects for Model-Based Coding of Facial Image Sequences", by G.Bozdagi, 
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A. Murat Tekalp and L Onural, IEEE Transactions on CSVT, Vol.4, No.3, pp.246-256, June 
1994, although more accurate and robust, is typically computationally burdensome. Therefore, 
a need exists for an approach that is more accurate than the former, but less computationally 
burdensome than the latter. 

In this particular context, the motion that is being tracked or coded is the movement of a 
head or face in a sequence of images. Having the ability to track this motion and coding it may 
be desirable for a number of reasons. As just a few examples, this may be desirable in video 
conferencing, where a camera at one end may transmits the appropriate motion or movement 
of face to a display at the other end. However, the communications channel by which this 
video conferencing may take place sometimes has a relatively low or limited bandwidth, so that 
only a limited amount of signal information may be communicated in real-time. 

An embodiment of a method of video coding a movement of human head or face from a 
sequence of images includes the following. A limited number of feature points are selected 
from an image of the face whose movement is to be video coded. Using at least two images or 
frames from the sequence, changes in the intensity of selected feature points, such as spatio- 
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temporal rates of change, are estimated. Using the feature points and the estimated rates, 
translation and rotation parameters of the face are then estimated. The estimated translation 
and rotation parameters are coded and/or transmitted across the communications channel. It is 
noted, of course that instead of communicating the coded signal information, it may, 
alternatively, be stored and read from memory for later use, or used in some other way other 
than by transmitting it. 

Although the invention is not limited in scope in this respect, in this particular 
embodiment, the face is coded from at least one of the images or frames by employing a three- 
dimensional (3D) based coding technique to produce what shall be referred to here as a 3D 
model. Movement of the face from at least two, typically sequential, images of the sequence is 
estimated using this 3D model of the face or head. In particular, as shall be described in more 
detail hereinafter, the movement of the face is estimated by treating the 3D model of the head 
as a rigid body in the sequence of images. 

In this embodiment, although the invention is not limited in scope in this respect, the 3D 
model applied comprises planar triangular patches. This illustrated, for example, in FIG. 1. As 

7 



FIG. 1 illustrates, these triangular patches, in this particular embodiment in accordance with the 
invention, are divided into two classes, one class in which local motion is more significant, such 
as, for example, the triangular patches covering eyes, eyebrows, or mouth, denoted here , 
and one class in which global motion is more significant, denoted here by the • g . FIG. 1 
illustrates the two classes of triangles, the shaded of triangles belonging to »i and unshaded 
triangles belonging to # g . 

In this embodiment, a limited number of feature points are selected from an image of 
the head. In this embodiment, enough feature points are selected from different triangular 
patches to obtain the desired amount of accuracy or robustness without being computationally 
burdensome. Furthermore, a weighting factor is assigned to each feature point, depending 
upon the class of triangular patch to which it belongs. The weighting factor assigned to a 
feature point selected from the i th triangular patch is given by the following relationship. 

W g , for alii £ « g 

W pF { 

W,, for alii G •, 
where W g is greater than W|. 



The weighting factors are used in the Least Mean Square estimation of the global motion 
parameters in this particular embodiment, as described in more detail later, and there, the facial 
regions contributing more to the global motion have more weighting factors than the ones 
predominantly contributing to local motion; however, the invention is not restricted in scope to 
this embodiment. For example, other estimation approaches other than Least Mean Square 
may be employed and other approaches to employing weighting may be employed, or, 
alternatively, weighting may not necessarily be employed in alternative embodiments. For this 
embodiment, the range of the weighting factors were determined from experimentation, 
although, again, the invention is not restricted in scope to this particular range of weights. Here, 
nonetheless, W g varies in the range of approximately 0.6 to approximately 0.9 and W, varies in 
the range of approximately 0.3 to approximately 0.1 . 

Once feature points are selected, the rate of change of intensity of the selected feature 
points is estimated from the sequence of images. It is noted that it takes at least two images to 
estimate a rate of change; however, in this embodiment a rate of change is calculated for each 
pair of immediately sequential images in the sequence . It is also noted that a distinguishing 
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feature of this approach is the selection of a limited number of feature points, thereby reducing 
the computational burden of this approach. 



The relationship between rate of change in intensity at the selected feature points and 
estimating the translation and rotation of the face is as follows. The gradient between two 
consecutive or immediately sequential frames is described as follows. 

IxkVxk + IykVyk + Itk =0 (1) 

where I X k, Iyk, and I T k are the rates of change at a selected pixel between the two 
frames k and (k+1) in the x-, y- and the temporal directions respectively and V X k , 
V Y k are optical flow fields in the x and y directions, respectively. The lx«and I Y k 
are determined by the intensity gradients of the neighboring pixels in the same 
frame and I T k is measured from the intensity gradient at substantially the same 
spatial position between consecutive frames. The equation is based on an 
assumption of brightness constancy for moving objects in the successive frames. 



Likewise, the formula for small motion of a rigid body is given by the following equation. 
V=RP + T (2) 
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where P is a three-dimensional position vector, vector V represents the velocity 
of a point on the rigid body, matrix R represents the angular velocity, and vector 
T represents the translation of the rigid body . R, the angular velocity, is given 
by the following 3-by-3 matrix 



rn 




T13 


r 2 i 


1*22 


1*23 


r 3 i 


1*32 


l"33 



where r n =c a c p -\, r n =c a s p s y -s a c y , r 13 = c a s p c y -c a s y , r 21 =s a s p , 

r 22 = s a s p s Y - c a c y - 1 , r n = s a s p c Y - c a s Y , r 31 = -s p , r 32 =c p s y , r 33 = c p c y - 1 and a, p, 

and Y are rotations about x, y, and z axes, respectively, and c and s denote the cosine and the 
sine of the angles. 

Under the assumption of orthographic projection of the human face, for this particular 
embodiment, V XK and Vyk are considered to be the optical flow fields with the z-diectional 
component assumed to be zero. The following linearized estimation equation may, therefore, be 
derived from equation (2) above by equating the x- and the y-directional components of the 
velocities and then using these relations in equation (1) to evaluate Itk as 
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H K = F K A 

where H K is -l TK , A is [rn r 12 r 13 r 2 i r 22 r 23 T x T Y ], and F« is 

I** 1 * yJ xk zj* Vj* yJyk z k i yk i xk i yk ] 

The global estimation parameters, here, the translation and rotation signal information, may be 
obtained by solving the previous equation using a Least Mean Square approach, although, of 
course, the invention is not limited in scope in this respect, and other approaches, such as least 
absolute value, may be employed. This is done for each feature point and then the values for a 
rotation and translation that give the least overall mean square error are employed, again, for 
this particular embodiment. 

FIG. 2 is a flowchart illustrating an embodiment 200 of a method of video coding the 
movement of a human face from a sequence of images in accordance with the invention, 
although, of course, this is just one example of an embodiment, and many others are possible 
within the scope of the present invention. At block 210, the face or head from a particular or 
selected image is approximated using triangular patches, such as illustrated, for example, in 
FIG. 1 . At block, 220, a limited number of feature points on the face are selected, in this 



particular embodiment, one feature point from each triangular patch, although, of course, the 
invention is not limited in scope in this respect. At block 230, for each pair of successive 
images or frames in the sequence of images, spatio-temporal rates of change in intensity at the 
selected feature points are estimated. At block 240, estimates of translation and rotation for the 
face are made using the feature points and the estimated spatio-temporal rates of change for 
each pair of successive images in this particular embodiment. At block 250, these estimates 
are then coded and transmitted across a transmission medium or communications channel so 
that at the far end, the estimates may be employed to reproduce or represent movement of a 
representation of the face. 

It will, of course, be understood that, although particular embodiments have just been 
described, the invention is not limited in scope to a particular embodiment or implementation. 
For example, one embodiment may be in hardware, whereas another embodiment may be in 
software. Likewise, an embodiment may be in firmware, or any combination of hardware, 
software, or firmware, for example. Likewise, although the invention is not limited in scope in 
this respect, one embodiment may comprise an article, such as a storage medium. Such a 
storage medium, such as, for example, a CD-ROM, or a disk, may have stored thereon 



instructions, which when executed by a system, such as a host computer or computing system 
or platform, or an imaging system, may result in a method of video coding the movement of a 
human face from a sequence of images in accordance with the invention, such as, for example, 
one of the embodiments previously described. Likewise, a hardware embodiment may 
comprise an imaging system including an imager and a computing platform, such as one 
adapted to perform or execute coding in accordance with the invention, for example. 

While certain features of the invention have been illustrated and detailed herein, many 
modifications, substitutions, changes and equivalents will now occur to those skilled in the art. 
It is, therefore, to be understood that the appended claims are intended to cover all such 
modifications and changes as fall within the true spirit of the invention. 
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Claims: 

1 . A method of video coding the movement of a face from a sequence of images 
comprising: 

selecting a limited number of feature points from an image of the face whose movement 
is to be video coded; 

estimating spatio-temporal rates of change in intensity at the selected feature points 
using at least two images from the sequence; 

estimating the translation and rotation of the face using the feature points and using the 
estimated spatio-temporal rates; and 

coding the estimated translation and rotation. 

2. The method of claim 1 , wherein prior to selecting the feature points, for a particular 
image, the face is approximated using triangular planar patches. 

3. The method of claim 2, wherein a feature point is selected from each planar triangular 
patch. 
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4. The method of claim 1, wherein the translation and rotation of the face are estimated 
from the feature points and from the estimated spatio-temporal rates using a least mean square 
estimation method. 

5. A method of video coding the movement of a head from a sequence of images 
comprising: 

coding the head from at least one of the images based, at least in part, on a limited 
number of selected feature points employing a three-dimensional based coding technique to 
produce a three-dimensional (3D) model; and 

estimating the movement of the head in other images of the sequence using the 3D 
model of the head. 

6. The method of claim 5, wherein the movement of the head is estimated by treating the 
3D model of the head as a rigid body in the sequence of images. 
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7. The method of claim 6, wherein the movement of the head is estimated as translations 
and rotations based at least in part on estimates of spatio-temporal rates of change in intensity 
at the selected feature points. 

8. A system comprising: 

an imager; and a computing platform; 

said imager and computing platform being coupled to communicate electronically; 
wherein said computing platform being adapted so that, in operation, the movement of a 
face from a sequence of images is coded by: 

selecting a limited number of feature points from an image of the face whose movement 
is to be video coded; 

estimating spatio-temporal rates of change in intensity at the selected feature points 
using at least two images from the sequence; 

estimating the translation and rotation of the face using the feature points and using the 
estimated spatio-temporal rates; and 

coding the estimated translation and rotation. 



9. The system of claim 8, wherein said computing platform is adapted so that, in operation, 
the movement of a face from a sequence of images is further coded by, prior to selecting the 
feature points, for a particular image, approximating the face using triangular planar patches. 

10. The system of claim 9, wherein said computing platform is adapted so that, in operation, 
the movement of a face from a sequence of images is further coded by selecting a feature point 
from each planar triangular patch. 

11. The system of claim 8, wherein said computing platform is adapted so that, in operation, 
the movement of a face from a sequence of images is further coded by estimating the 
translation and rotation of the face from the feature points and from the estimated spatio- 
temporal rates using a least mean square estimation method. 

12. A system comprising: 

an imager; and a computing platform; 

said imager and computing platform being coupled to communicate electronically; 
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wherein said computing platform being adapted so that, in operation, the movement of a head 
from a sequence of images is represented by: 

coding the head from at least one of the images based, at least in part, on a limited 
number of selected feature points employing a three-dimensional based coding technique to 
produce a three-dimensional (3D) model; and 

estimating the movement of the head in other images of the sequence using the 3D 
model of the head. 

13. The system of claim 12, wherein said computing platform is adapted so that, in 
operation, the movement of the head in other images of the sequence is estimated by treating 
the 3D model of the head as a rigid body in the sequence of images. 

14. The system of claim 13, wherein said computing platform is adapted so that, in 
operation, the movement of the head is estimated as translations and rotations based at least 
in part on estimates of spatio-temporal rates of change in intensity at the selected feature 
points. 



15. An article comprising: a storage medium, said storage medium having stored thereon 
instructions, said instructions, when executed by a computing platform, resulting in the 
movement of a face from a sequence of images being coded by: 

selecting a limited number of feature points from an image of the face whose movement 
is to be video coded; 

estimating spatio-temporal rates of change in intensity at the selected feature points 
using at least two images from the sequence; 

estimating the translation and rotation of the face using the feature points and using the 
estimated spatio-temporal rates; and 

coding the estimated translation and rotation. 

16. The article of claim 15, wherein said instructions, when executed, further result in, prior 
to selecting the feature points, for a particular image, approximating the face using triangular 
planar patches. 

17. The article of claim 16, wherein said instructions, when executed, further result in 
selecting a feature point from each planar triangular patch. 
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18. The article of claim 15, wherein said instructions, when executed, further result in, 
estimating the translation and rotation of the face from the feature points and from the 
estimated spatio-temporal rates using a least mean square estimation method. 

1 9. An article comprising: a storage medium, said storage medium having stored thereon 
instructions, said instructions, when executed by a computing platform, result in the movement 
of a head from a sequence of images being represented by: 

coding the head from at least one of the images based, at least in part, on a limited 
number of selected feature points employing a three-dimensional based coding technique to 
produce a three-dimensional (3D) model; and 

estimating the movement of the head in other images of the sequence using the 3D 
model of the head. 

20. The article of claim 19, wherein said instructions, when executed, further result in the 
the movement of the head in other images of the sequence being estimated by treating the 3D 
model of the head as a rigid body in the sequence of images. 
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21 . The article of claim 20, wherein said instructions, when executed, result in the 
movement of the head being estimated as translations and rotations based at least in part on 
estimates of spatio-temporal rates of change in intensity at the selected feature points. 



Abstract 

Briefly, in accordance with one embodiment, a technique for coding the movement of a 
head or face from a sequence of images is disclosed. A variety of potential alternative 
embodiments are discussed. 
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Full Name of Sole/First Inventor (given name, family name) Tinku Acharya 



Inventor's Signature Date 



Residence Citizenship 

(City , State) (Country) 

P. O. Address 
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Full Name of Second/Joint Inventor (given name, family name) Somnath Sengupta 



Inventor's Signature Date 



Residence Citizenship 

(City , State) (Country) 

P. (X Address 



Full Name of Third/Joint Inventor (given name, family name) Rama A. Suryanarayana 



Inventor's Signature Date 



Residence Citizenship 

(City , State) (Country) 

P. O. Address 



Full Name Of Fourth/ Joint Inventor (given name, family name) 



Inventor's Signature Date 



Residence Citizenship 

(City , State) (Country) 

R O. Address 



Full Name of Fifth/ Joint Inventor (given name, family name) 



Inventor's Signature Date 



Residence Citizenship 

(City, State) (Country) 

P, (X Address 
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Full Name Of Sixth/Joint Inventor (given name, family name) 



Inventor's Signature Date 



Residence Citizenship 

(City , State) (Country) 

P. O. Address 



Full Name of Seventh/Joint Inventor (given name, family name) 



Inventor's Signature Date 



Residence Citizenship 

(City , State) (Country) 

P. O. Address 



Full Name of Eighth/Joint Inventor (given name, family name) 



Inventor's Signature Date 



Residence Citizenship 

(City , State) (Country) 

P. O. Address 



Full Name Of Ninth/Joint Inventor (given name, family name) 



Inventor's Signature Date 



Residence ^ Citizenship 

(City , State) (Country) 

P. O. Address 
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Full Name Of Tenth/ Joint Inventor (given name, family name) 



Inventor's Signature Date 



Residence Citizenship 

( City , State) (Country) 

P. O. Address 



Full Name of Eleventh/Joint Inventor (given name, family name) 



Inventor's Signature Date 



Residence Citizenship 

(City , State) (Country) 

P. O. Address 
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APPENDIX A 



I hereby appoint BLAKELY, SOKOLOFF, TAYLOR & ZAFMAN LLP, a firm including: William E. Alford, Reg. 
No. 37,764; Farzad E. Amini, Reg. No. 42,261 ; Amy M. Armstrong, Reg. No. 42,265; Aloysius T. C. AuYeung, Reg. No. 
35,432; William Thomas Babbitt, Reg. No. 39,591; Carol F. Barry, Reg. No. 41,600; Jordan Michael Becker, Reg. No. 
39,602; Bradley J. Bereznak, Reg. No. 33,474; Michael A. Bernadicou, Reg. No. 35,934; Roger W. Blakely, Jr., Reg. No. 
25,831; Gregory D. Caldwell, Reg. No. 39,926; Ronald C. Card, Reg. No. 44,587; Thomas M. Coester, Reg. No. 39,637; 
Donna Jo Coningsby, Reg. No. 41,684; Michael Anthony DeSanctis, Reg. No. 39,957; Daniel M. De Vos, Reg. No. 37,813; 
Robert Andrew Diehl, Reg. No. 40,992; Matthew C. Fagan, Reg. No. 37,542; TarekN. Fahmi, Reg. No. 41,402; George L. 
Fountain, Reg. No. 36,374; Paramita Ghosh, Reg. No. 42,806; James Y. Go, Reg. No. 40,621; James A. Henry, Reg. No. 
41,064; WillmoreF. Holbrow IH, Reg. No. 41,845; Sheryl Sue Holloway, Reg. No. 37,850; George W Hoover II, Reg. No. 
32,992; Eric S. Hyman, Reg. No. 30,139; William W. Kidd, Reg. No. 31,772; Sang Hui Kim, Reg. No. 40,450; Eric T. King, 
Reg. No. 44,188; Jason K. Klindtworth, Reg, No. P47,21 1; Erica W. Kuo, Reg. No. 42,775; Michael J. Mallie, Reg. No. 
36,591; Paul A. Mendonsa, Reg. No. 42,879; Clive D. Menezes, Reg. No. 45,493; Chun M. Ng, Reg. No. 36878; Thien T. 
Nguyen, Reg. No. 43,835; Thinh V. Nguyen, Reg. No. 42,034; Dennis A. Nicholls, Reg. No. 42,036; Lisa A. Norris, Reg. No. 
44,976; Daniel E. Ovanezian, Reg. No. 41,236; William F. Ryann, Reg. No. 44,313; James H. Salter, Reg. No. 35,66S; 
William W. Schaal, Reg. No. 39,018; James C, Scheller, Reg. No. 31,195; Jeffrey S. Smith, Reg. No. 39,377; Maria 
McCormack Sobrino, Reg. No. 31,639; Stanley W. Sokoloff, Reg. No. 25,128; Judith A. Szepesi, Reg. No. 39,393; Vincent P. 
Tassinari, Reg. No. 42,179; Edwin H. Taylor, Reg. No. 25,129; Joseph A. Twarowski, Reg. No. 42,191; Lester J. Vincent, 
Reg. No. 31,460; Glenn E. Von Tersch, Reg. No. 41,364; John Patrick Ward, Reg. No. 40,216; Charles T. J. Weigell, Reg. 
No. 43,398; James M. Wu, Reg. No. 45,241; Steven D. Yates, Reg. No. 42,242; and Norman Zafman, Reg. No. 26,250; my 
attorneys; and Andrew C. Chen, Reg. No. 43,544; Justin M. Dillon, Reg. No. 42,486; and John F. Travis, Reg. No. 43,203; my 
patent agents, of BLAKELY, SOKOLOFF, TAYLOR & ZAFMAN LLP, with offices located at 12400 Wilshire Boulevard, 7th 
Floor, Los Angeles, California 90025, telephone (714) 557-3800, and Alan K. Aldous, Reg. No. 31,905; Robert D. Anderson, 
Reg. No. 33,826; Joseph R. Bond, Reg. No. 36,458; Richard C. Calderwood, Reg. No. 35,468; Jeffrey S. Draeger, Reg. No. 
41,000; Cynthia Thomas Faatz, Reg No. 39,973; Sean Fitzgerald, Reg. No. 32,027; John N, Greaves, Reg. No. 40,362; Seth Z. 
Kalson, Reg. No. 40,670; David J. Kaplan, Reg. No. 41,105; Charles A. Mirho, Reg. No. 41,199; Leo V. Novakoski, Reg. 
No. 37,198; Naomi Obinata, Reg. No. 39,320; Thomas C. Reynolds, Reg. No. 32,488; Kenneth M. Seddon, Reg. No. 43,105; 
Mark Seeley, Reg. No. 32,299; Steven P. Skabrat, Reg. No. 36,279; Howard A. Skaist, Reg. No. 36,008; Steven C. Stewart, 
Reg. No, 33,555; Raymond J. Werner, Reg. No. 34,752; Robert G. Winkle, Reg. No. 37,474; and Charles K. Young, Reg. No. 
39,435; my patent attorneys, and Thomas Raleigh Lane, Reg. No. 42,781; Calvin E. Wells; Reg. No. P43,256, Peter Lam, Reg. 
No. 44,855; and Gene I. Su, Reg. No. 45,140; my patent agents, of INTEL CORPORATION; and James R. Them, Reg. No. 
31,710, my patent attorney; with full power of substitution and revocation, to prosecute this application and to transact all 
business in the Patent and Trademark Office connected herewith. 
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